Summary

Makes the WARC-repackaging command's range-job stage pluggable behind a RangeJobSource
abstraction, and renames the command warc_by_cdx → repackage (it now repackages WARC ranges
from several sources, not just CDX).

Range jobs (a WARC file + byte range) can now come from:

Athena and DuckDB share one SQL core (sources/sql_base.py); they differ only in the FROM clause
and execution. The pipeline orchestrator now owns queueing, the record limit, counting, and
stop-sentinel emission (in a finally) — which also fixes a latent bug where a source raising
mid-run left the WARC readers hung forever. Each source owns its own stage-1 client/connection;
WARCFilter manages only the read/write S3 clients.

Base branch: this PR targets feat/warc-by-cdx (PR #54), not main.

CLI usage

Global flags (--crawl, --limit) come before the repackage subcommand; source flags come after.

CDX files (default)

cdxt repackage \
    --target-source cdx \
    --cdx-path filtered_CC-MAIN-2024-30.cdx.gz \
    --prefix ./out/EXAMPLE \
    --warc-download-prefix https://data.commoncrawl.org
# multiple indices via glob:
cdxt repackage --target-source cdx --cdx-path s3://bucket/cdx/ --cdx-glob '*.cdx.gz' --prefix ./out/EXAMPLE

SQL — Athena (guided by hostnames/domains, pruned by crawl)

cdxt --crawl CC-MAIN-2026-17 repackage \
    --target-source sql --engine athena \
    --hostnames commoncrawl.org www.commoncrawl.org \
    --athena-database ccindex \
    --athena-s3-output s3://my-bucket/athena-results/ \
    --prefix s3://my-bucket/out/CC \
    --warc-download-prefix s3://commoncrawl

The guided filter matches on url_host_name (exact host) via --hostnames and/or
url_host_registered_domain via --domains (which also covers subdomains). They can be combined
(predicates are OR-ed):

# every host under the example.com registered domain, plus one exact extra host
cdxt --crawl CC-MAIN-2026-17 repackage \
    --target-source sql --engine duckdb \
    --domains example.com \
    --hostnames blog.example.org \
    --prefix ./out/EX

SQL — DuckDB (reads the public parquet directly; no Athena charge)

cdxt --crawl CC-MAIN-2026-17 repackage \
    --target-source sql --engine duckdb \
    --hostnames commoncrawl.org \
    --prefix ./out/CC
# requires the optional dependency:  pip install cdx_toolkit[duckdb]

SQL — raw query (power users)

The query must SELECT warc_filename, warc_record_offset, warc_record_length:

cdxt repackage \
    --target-source sql --engine athena \
    --query "SELECT warc_filename, warc_record_offset, warc_record_length
             FROM ccindex
             WHERE subset = 'warc'
               AND crawl = 'CC-MAIN-2026-17'
               AND url_host_registered_domain = 'commoncrawl.org'
               AND content_mime_type = 'application/pdf'
             LIMIT 5000" \
    --athena-database ccindex \
    --athena-s3-output s3://my-bucket/athena-results/ \
    --prefix ./out/PDFS \
    --confirm-cost
# or load it from a file with --query-file ./my_query.sql

CSV (consume a previously materialized range-jobs file)

cdxt repackage \
    --target-source csv \
    --csv-path ranges.csv \
    --prefix ./out/CC \
    --warc-download-prefix https://data.commoncrawl.org

Cost guard

SQL index scans bill by data scanned (Athena: ~$5/TB), so repackage prompts before a potentially
expensive query. It runs without a prompt only when the query is restricted to ≤ 10 crawls.
Otherwise (no --crawl → all crawls, > 10 crawls, or a raw --query whose pruning can't be
verified) it asks for confirmation; in a non-interactive shell it aborts unless --confirm-cost is
passed. cdx and csv sources never prompt.

WARNING  This duckdb query is not restricted to specific crawls ... and may scan ALL crawls.
Index SQL scans can be expensive (Athena bills per TB scanned). Proceed? [y/N]

Range-jobs CSV (materialization)

Any source can write its resolved range jobs to a CSV with --range-jobs-output. Add --no-fetch
to only produce the CSV (cheap — no WARC download), then consume it later. This decouples the
(possibly expensive) index query from extraction, makes runs reproducible/shareable, and lets the
whole reader/writer pipeline be tested without AWS.

# 1) produce the ranges once (cheap, single-crawl query, no WARC fetch)
cdxt --crawl CC-MAIN-2026-17 repackage \
    --target-source sql --engine duckdb --hostnames commoncrawl.org \
    --range-jobs-output ranges.csv --no-fetch

# 2) fetch + repackage from the CSV, as many times as you like
cdxt repackage --target-source csv --csv-path ranges.csv \
    --warc-download-prefix https://data.commoncrawl.org --prefix ./out/CC

CSV formats

Default (relative filename) — the consumer prepends --warc-download-prefix:

warc_filename,warc_record_offset,warc_record_length
crawl-data/CC-MAIN-2026-17/segments/1234567890.12/warc/CC-MAIN-20260101000000-20260101010000-00000.warc.gz,111440525,9754
crawl-data/CC-MAIN-2026-17/segments/1234567890.12/warc/CC-MAIN-20260101000000-20260101010000-00001.warc.gz,98231,12044

Self-contained (--csv-self-contained) — full URLs, used as-is (no prefix needed). The reader
auto-detects the mode from the header (warc_url vs warc_filename); if both are present it warns
and uses warc_url:

warc_url,warc_record_offset,warc_record_length
https://data.commoncrawl.org/crawl-data/CC-MAIN-2026-17/segments/1234567890.12/warc/CC-MAIN-...-00000.warc.gz,111440525,9754
s3://commoncrawl/crawl-data/CC-MAIN-2026-17/segments/1234567890.12/warc/CC-MAIN-...-00001.warc.gz,98231,12044

.tsv/.tsv.gz inputs are read as tab-delimited; .gz inputs are decompressed.

Extra columns from a raw --query — any column a raw --query/--query-file SELECTs beyond the
required warc_filename, warc_record_offset, warc_record_length is carried through to the
range-jobs CSV (handy for later analysis), e.g. adding content_languages:

warc_filename,warc_record_offset,warc_record_length,content_languages
crawl-data/CC-MAIN-2026-17/segments/1234567890.12/warc/CC-MAIN-...-00000.warc.gz,111440525,9754,eng

The fetch-job columns are named warc_* precisely so they never collide with the index's own
columns (e.g. the page-URL column url flows through as an ordinary extra column, distinct from
warc_url). These extra columns don't affect fetching.

Fetch-time ordering

Range jobs are sorted by (warc_filename, warc_record_offset) by default, grouping records of the
same WARC file with ascending offsets for better S3 range-read locality (and to enable coalescing
adjacent ranges later). This is applied as an ORDER BY on a guided SQL query (Athena/DuckDB) and as
an in-memory sort when loading a CSV source. A raw --query/--query-file is left untouched (order
it yourself), and cdx sources keep their native SURT order. Pass --no-sort-ranges to disable —
useful for an already-sorted or very large CSV (the CSV sort buffers all rows in memory), or to
preserve a source's original order.

Optional dependency

DuckDB is optional: pip install cdx_toolkit[duckdb]. Without it, the other sources work unchanged;
selecting --engine duckdb raises a clear error.

Testing

• Unit (no AWS): SQL builder per engine (incl. --domains / combined host+domain),
  escape_sql_literal injection rejection, CSV
  reader/writer round-trip in both modes + header auto-detection, the make_source factory
  validation, the confirm_cost guard, --no-fetch, and a regression test that the producer still
  releases readers (_STOP) when a source raises.
• CSV round-trip e2e (offline produce from the CDX fixture + HTTP consume), both modes.
• Gated e2e (skipped in CI): Athena and DuckDB querying CC-MAIN-2026-17 / commoncrawl.org
  (single partition — cheap, never an all-crawls scan), including a DuckDB --domains run.

Verified locally: full suite 205 passed; DuckDB e2e ran live and passed; all 6 S3
aioboto3 read/write tests pass; flake8 clean. (Athena gated test skips where Athena permissions
aren't available; its code path is covered by unit tests and shares the orchestration/fetch path
proven by the DuckDB e2e.)